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EXECUTIVE  SUMMARY 


PROBLEM 

The  Department  of  Defense  (DoD),  other  Government  agencies,  academia,  and  private 
industry  are  required  to  provide  education  and  training  on  an  increasingly  varied  and  complex 
range  of  tasks  and  skills,  with  a  dwindling  amount  of  resources.  To  meet  this  challenge,  the  DoD 
and  the  White  House  Office  of  Science  and  Technology  Policy  (OSTP)  launched  the  Advanced 
Distributed  Learning  (ADL)  Initiative  in  November  1997.  The  goal  of  the  ADL  Initiative  is  to 
ensure  access  to  high  quality  education  and  training  materials  that  can  be  tailored  to  individual 
learner  needs  and  made  available  whenever  and  wherever  they  are  required 

OBJECTIVE 

This  objective  of  this  report  is  to  provide  a  theory-based  ADL  research  agenda.  This  research 
agenda  must  identify  critical  research  areas  and  issues  to  be  explored  if  the  goals  of  the  ADL 
Initiative  are  to  be  achieved.  Furthermore,  within  each  research  area,  methods  and  techniques 
must  be  identified  that  will  efficiently  generate  the  data  required  to  achieve  these  goals. 

APPROACH 

An  overview  of  system  concepts  and  methods  is  applied  to  identify  and  discuss  critical  ADL 
research  areas  and  issues.  Moreover,  these  system  concepts  are  used  to  argue  that  a  system 
perspective  can  be  beneficially  applied  to  ADL  system  development.  The  adoption  of  a  system 
perspective  leads  individuals  to  focus  on  the  complex  interactions  between  and  among  the  many 
varied  systems,  which  affect  the  evolution  of  ADL  capabilities.  System  theory-based  tools  and 
methods  are  identified  for  each  research  area. 

RESULTS 

Five  system  considerations  (Churchman,  1968)  were  used  to  discuss  ADL  from  a  system 
perspective.  These  considerations  are:  understand  the  total  ADL  system  objectives,  understand 
the  ADL  system's  environment,  understand  the  resources  available,  understand  the  ADL 
subsystem  components,  and  understand  the  management  of  the  ADL  system.  Several  research 
areas  are  discussed:  application  of  system  concepts  and  approaches  to  ADL,  cross-level 
generalizations,  identification  of  system  relationships,  ADL  goals  and  needs  analysis,  ADL 
development  guidelines,  evaluation  of  ADL  systems,  and  configuration  management  of  ADL 
systems. 

CONCLUSIONS  AND  RECOMMENDATIONS 

Guided  by  a  system  perspective,  the  application  of  system  tools  and  concepts  can  improve 
our  ability  to  develop  effective  and  economical  ADL  systems.  Adoption  of  the  research  agenda 
presented  in  the  paper  is  a  first  step  in  generating  the  data  necessary  to  reach  this  goal. 
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INTRODUCTION 


PROBLEM 

As  we  begin  the  twenty-first  century,  the  Department  of  Defense,  other  Government  agencies, 
academia,  and  private  industry  are  required  to  provide  education  and  training  on  an  increasingly 
varied  and  complex  range  of  tasks  and  skills,  with  a  dwindling  amount  of  resources.  To  meet 
this  challenge,  the  Department  of  Defense  (DoD)  and  the  White  House  Office  of  Science  and 
Technology  Policy  (OSTP)  launched  the  Advanced  Distributed  Learning  (ADL)  Initiative  in 
November  1997. 

The  goal  of  the  ADL  Initiative  is  to  ensure  access  to  high  quality  education  and  training 
materials  that  can  be  tailored  to  individual  learner  needs  and  made  available  whenever  and 
wherever  they  are  required.  ADL  has  the  potential  for:  reducing  education  and  training  costs, 
providing  instruction  on  time  and  on  demand,  increasing  access  to  instruction,  increasing 
availability  of  refresher  training  and  job  performance  support,  and  increasing  the  retention  of 
learning.  As  part  of  the  ADL  Initiative,  the  Office  of  the  Deputy  Secretary  of  Defense 
(Readiness)  established  the  ADL  Co-Laboratory  Network.  The  ADL  Co-Laboratory,  located  in 
Alexandria,  VA,  coordinates  development  of  common  standards  for  ADL  and  promotes 
interagency  coordination  and  collaboration  on  ADL  efforts.  The  Joint  ADL  Co-Laboratory, 
located  in  Orlando,  FL,  develops  techniques  and  guidance  to  implement  ADL  in  the  military 
services.  The  Academic  Co-Laboratory,  at  the  University  of  Wisconsin,  Madison,  WI,  focuses 
primarily  on  ADL  efforts  in  academic  institutions. 

The  shift  of  even  a  portion  of  DoD  education  and  training  from  traditional  delivery  methods 
to  ADL  is  a  huge  undertaking.  For  example,  there  are  over  190  schoolhouses  under  the 
cognizance  of  the  Chief  of  Naval  Education  and  Training  (CNET).  These  schoolhouses  provide 
instruction  to  approximately  80,000  students  per  year.  Plans  to  reduce  schoolhouse  infrastructure 
require  the  conversion  of  approximately  3400  courses  to  some  form  of  ADL  (Slater,  2001).  This 
is  only  a  small  portion  of  the  ongoing  ADL  efforts  in  the  military  services.  Similar  large-scale 
efforts  are  also  ongoing  in  academia  and  industry. 

OBJECTIVE 

The  objective  of  this  report  is  to  provide  a  theoretical  foundation  for  an  ADL  research  agenda, 
which  will  help  reduce  the  fragmentation  often  found  in  the  development  and  implementation  of 
instructional  systems.  Many  instructional  developers  (e.g.,  Merrill,  Li,  &  Jones,  1990)  have 
noted  that  a  major  limitation  of  instructional  development  approaches  is  the  failure  to  integrate 
the  various  phases  of  development  and  implementation.  This  lack  of  instructional  system 
integration  has  been  shown  to  compromise  their  effectiveness  (e.g.,  Caro,  1977;  Hritz  &  Purifoy, 
1980;  Iffland  &  Whiteside,  1977;  Kane  &  Holman,  1982).  On  the  other  hand,  evidence  also 
exists  that  enhanced  system  integration  can  improve  instructional  effectiveness  (e.g.,  Martin  & 
Rose,  1988;  Rose  &  Martin,  1988).  As  with  any  other  instructional  approach,  ADL  will  not  be 
optimally  successful  unless  it  achieves  a  higher  degree  of  system  integration.  A  more  in-depth 
understanding  of  system  concepts  can  help  the  individuals  engaged  in  ADL  research  identify  the 
critical  issues  that  must  be  addressed  to  ensure  more  effective  ADL  instructional  products. 
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ORGANIZATION  OF  THE  REPORT 

The  remainder  of  the  report  includes  four  major  sections.  The  next  section  provides  an 
overview  of  system  theory  and  system  concepts.  The  following  section  includes  a  discussion  of 
ADL  systems  in  the  context  of  Instructional  System  Development.  System  approaches  are 
applied  to  ADL  in  the  third  section.  The  fourth  section  is  a  discussion  of  seven  ADL  research 
areas  and  the  system  tools  and  methods,  which  can  be  applied  in  each  area.  The  report  concludes 
with  a  summary  and  recommendations  section. 
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OVERVIEW  OF  SYSTEM  CONCEPTS 

DEFINITION  OF  SYSTEMS 

Systems  have  been  defined  in  many  different  ways  (e.g.,  Churchman,  1968;  Koestler,  1969; 
Laszlo,  1972;  Miller,  1978;  Ruesch,  1969;  van  Gigch,  1978;  Weiss,  1969;  &  Wright,  1989).  For 
the  purposes  of  this  report,  the  following  major  concepts  will  collectively  serve  as  our  definition 
of  systems: 

1)  Systems  are  bounded  sets  of  interrelated  parts,  such  that  changes  in  one  part  will  cause 
changes  in  the  other  parts. 

2)  Systems  act  to  maintain  their  internal  consistency  even  in  the  face  of  external  changes. 

3)  The  emergent  qualities  of  a  system  cannot  be  explained  by  mere  addition  of  parts;  rather, 
relations  and  interactions  within  the  system  must  explain  these  qualities. 

4)  Systems  are  always  embedded  in  a  hierarchy  of  subsystems  and  suprasystems,  and 
interactions  occur  between  system  levels  at  all  times.  Every  system  is  a  component  (subsystem) 
of  a  larger  system.  The  larger  system,  which  includes  the  system  of  interest,  is  called  a 
suprasystem.  A  simplified  depiction  of  this  relationship  is  shown  in  Figure  1.  The  system 


Level  I 


Level  II 


Level  III 


Suprasystem 


Suprasystem 


Additional  Levels 
of  Subsystems 


Figure  1.  Relationship  of  Sub-  and  Suprasystems 
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shown  at  Level  I  is  made  up  of  three  subsystems.  A  larger  view  of  these  subsystems  is  labeled 
Level  II.  One  of  these  Level  II  systems  includes  five  subsystems,  which  are  shown  in  greater 
detail  on  Level  III.  Additional  sub-  and  suprasystems  could  be  depicted  in  both  directions. 

5)  A  system,  when  viewed  from  its  own  level,  seems  autonomous  and  it  appears  to  exert 
control  over  its  subsystems.  On  the  other  hand,  when  viewed  from  the  level  of  the  suprasystem, 
it  appears  to  be  under  the  suprasystem's  control.  A  problem  can  arise  when  a  subsystem  attempts 
to  reach  a  goal  that  is  in  conflict  with  those  of  its  suprasystem.  No  matter  how  much  effort  is 
expended  by  the  subsystem,  the  suprasystem's  goals  will  dominate.  Furthermore,  some  systems 
may  be  a  subsystem  of  more  than  one  suprasystem,  as  shown  in  Figure  2.  This  can  cause  even 
more  problems,  especially  if  the  goals  of  the  multiple  suprasystems  conflict. 


Alternate 

Suprasystem 


Level  III 


■o©',  Level  L 


Suprasystem 


Level  II 
Suprasystem 


*’•  Additional  Levels 
of  Subsystems 


Figure  2.  A  Subsystem  as  Part  of  More  than  One  Suprasystem 

Hays  (1992)  argued  that  system  definitions  and  considerations  apply  to  training  systems.  This 
report  argues  that  they  also  apply  to  ADL  systems.  In  later  sections,  selected  system  concepts 
will  be  applied  to  the  analysis  of  ADL  systems. 

SYSTEM  APPROACHES 

Systems  have  been  studied  since  Aristotle  began  to  examine  living  organisms  (Gaines,  1978). 
Psychophysicists  examine  sensation  and  perception  as  interactions  among  various  systems  within 
the  organism  (Gibson,  1966).  Some  of  the  more  widely  known  system  approaches  include: 

•  Cybernetics,  which  is  based  on  the  theory  of  communication  and  control  and 
emphasizes  information  transfer  and  feedback  (Wiener,  1948). 
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•  Graph  and  Net  theory,  which  focuses  on  the  structural  or  topological  properties  of 
systems  (Cartwright  &  Harary,  1956;  Haraiy  &  Schwenk,  1974). 

•  Other  system  approaches,  such  as  information  theory  (Shannon  &  Weaver,  1949), 
automata  theory  (von  Neumann,  1956),  and  fuzzy  set  theory  (Bellman  &  Zadeh,  1970). 

All  of  these  approaches  share  one  major  characteristic;  they  study  the  properties  of  systems, 
viewed  as  wholes,  interlinked  with  other  systems  and  explained  by  system  laws.  These  and  other 
system  theories  have  been  grouped  under  a  unifying  approach  called  General  Systems  Theory. 

General  Systems  Theory 

General  Systems  Theory  (GST)  arose  as  an  interdisciplinary  effort  to  understand  the  complex 
interactions  of  biological  systems  and  subsystems  (von  Bertalanffy,  1968;  1969).  The 
interdisciplinary  nature  of  GST  has  fostered  synergistic  relationships  that  can  advance  scientific 
understanding. 

GST  is  more  than  a  set  of  theories.  It  is  a  paradigm  shift  (Kuhn,  1962)  in  the  way  one  views 
and  orders  the  world  (Boulding,  1964;  Laszlo,  1972).  The  system  perspective  prompts 
individuals  to  look  beyond  their  own  subsystem  to  the  interactions  between  subsystems  and 
suprasystems.  GST  is  concerned  with  all  types  of  systems.  A  branch  of  GST  that  is  "concerned 
with  a  special  subset  of  systems,  the  living  ones,  "  (Miller,  1978,  p.  9)  is  called  Living  Systems 
Theory  (LST). 

Living  Systems  Theory 

LST  (Miller,  1978)  applies  the  system  perspective  to  all  living  systems  ranging  from  cells  to 
societies.  Three  of  the  major  theoretical  assumptions  of  LST  are: 

1)  There  are  19  critical  subsystem  processes,  which  each  living  system  must 
accomplish,  or  have  some  other  system  accomplish  for  it,  if  it  is  to  remain  viable.  Table  1 
summarizes  these  subsystem  processes.  The  labels  for  the  19  subsystem  processes  were 
specifically  chosen  to  be  general  so  they  could:  a)  "be  as  acceptable  as  possible  when  applied  at 
all  levels  and  to  all  types  of  living  systems"  (Miller,  1978,  p.  27);  and  b)  be  neutral,  "not  be 
associated  exclusively  with  any  type  or  level  or  system,  with  biological  or  social  science,  with 
any  discipline,  or  with  any  particular  school  or  theoretical  point  of  view"  (Miller,  1978,  p.  27). 

2)  Systems  at  all  levels  (from  the  cell  to  the  society)  have  the  same  19  critical 
subsystems.  However,  at  more  complex  levels,  the  critical  subsystem  processes  are  not  tied  to 
single  subsystems  in  a  one-to-one  fashion.  Rather,  the  functions  may  be  dispersed  among 
various  subsystems  in  a  process  labeled  "shred-out."  "Each  process  is  broken  down  into  multiple 
subprocesses  which  are  mapped  upon  multiple  structures,  each  of  which  becomes  specialized  for 
carrying  out  a  subprocess"  (Miller,  1978,  p.  26).  In  certain  cases,  via  shred-out,  the  critical 
processes  are  dispersed  to  various  components  of  the  system,  to  components  of  its  suprasystem, 
or  even,  to  artifacts  created  for  specific  functions.  For  example,  in  a  society,  the  distribution 
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Table  1 

The  19  Critical  Subsystems  of  a  Living  System 


Subsystems  Which  Process  Both  Matter-Energy  and  Information 

1 .  Reproducer,  the  subsystem  which  is  capable  of  giving  rise  to  other  systems  similar  to  the  one  it  is  in. 

2.  Boundary,  the  subsystem  at  the  perimeter  of  a  system  that  holds  together  the  components  which  make  up  the  system, 
protects  them  from  environmental  stresses,  and  excludes  or  permits  entry  to  various  sorts  of  matter-energy  and  information. 

Subsystems  Which  Process  Matter-Energy 

Subsystems  Which  Process  Information 

3.  Ingester,  the  subsystem  which  brings  matter-energy 
across  the  system  boundary  from  the  environment. 

1 1 .  Input  transducer,  the  sensory  subsystem  which  brings 
markers  bearing  information  into  the  system,  changing  them  to 
other  matter-energy  forms  suitable  for  transmission  within  it. 

12.  Internal  transducer,  the  sensory  subsystem  which 
receives,  from  subsystems  or  components  within  the  system, 
markers  bearing  information  about  significant  alterations  in 
those  subsystems  or  components,  changing  them  to  other 
matter-energy  forms  of  a  sort  which  can  be  transmitted  within  it. 

4.  Distributor,  the  subsystem  which  carries  inputs  from 
outside  the  system  around  the  system  to  each  component. 

13.  Channel  and  net,  the  subsystem  composed  of  a  single 
route  in  physical  space,  or  multiple  interconnected  routes,  by 
which  markers  bearing  information  are  transmitted  to  all  parts 
of  the  system. 

5.  Converter,  the  subsystem  which  changes  certain  inputs 
to  the  system  into  forms  more  useful  for  the  special  processes 
of  that  particular  system. 

14.  Decoder,  the  subsystem  which  alters  the  code  of 
information  input  to  it  through  the  input  transducer  or  internal 
transducer  into  a  "private”  code  that  can  be  used  internally  by 
the  system. 

6.  Producer,  the  subsystem  which  forms  stable 
associations  that  endure  for  significant  periods  among  matter- 
energy  inputs  to  the  system  or  outputs  from  its  converter,  the 
materials  synthesized  being  for  growth,  damage  repair,  or 
replacement  of  components  of  the  system,  or  for  providing 
energy  for  moving  or  constituting  the  system's  outputs  of 
products  or  information  markers  to  its  suprasystem. 

1 5.  Associator,  the  subsystem  which  carries  out  the  first  stage 
of  the  learning  process,  forming  enduring  associations  among 
items  of  information  in  the  system. 

7.  Matter-energy  storage,  the  subsystem  which  retains  in 
the  system  for  different  periods  of  time,  deposits  of  various 
sorts  of  matter-energy. 

1 6.  Memory,  the  subsystem  which  carries  out  the  second 
stage  of  the  learning  process,  storing  various  sorts  of 
information  in  the  system  for  different  periods  of  time. 

17.  Decider,  the  executive  subsystem  which  receives 
information  inputs  from  all  other  subsystems  and  transmits  to 
them  information  outputs  that  control  the  entire  system. 

1 8.  Encoder,  the  subsystem  which  alters  the  code  of 
information  input  to  it  from  other  information  processing 
subsystems,  from  a  "private"  code  used  internally  by  the  system 
into  a  "public"  code  which  can  be  interpreted  by  other  systems 
in  its  environment. 

8.  Extruder,  the  subsystem  which  transmits  matter-energy 
out  of  the  system  in  the  form  or  products  or  wastes. 

9.  Motor,  the  subsystem  which  moves  the  system  or  parts  of 
it  in  relation  to  part  or  all  of  its  environment  in  relation  to 
each  other. 

19.  Output  transducer,  the  subsystem  which  puts  out 
markers  bearing  information  from  the  system,  changing  markers 
within  the  system  into  other  matter-energy  forms  which  can  be 
transmitted  over  channels  in  the  system's  environment. 

10.  Supporter,  the  subsystem  which  maintains  the  proper 
spatial  relationships  among  components  of  the  system,  sot 
that  they  can  interact  without  weighting  each  other  down  or 
crowding  each  other. 

Note.  From  Living  Systems,  by  J.  G.  Miller,  p.  3.  Copyright  1978  by  McGraw-Hill,  Inc. 
Adapted  by  permission. 
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function  (see  Table  1)  is  accomplished  by  a  wide  range  of  individuals,  such  as  teamsters  and 
postal  workers,  using  a  variety  of  tools  like  sorting  machines,  trucks,  and  airplanes. 

3)  Since  all  systems  share  theses  19  subsystem  processes,  it  is  possible  to  identify  cross¬ 
level  generalizations  that  assist  in  the  study  of  living  systems.  For  example,  Miller  (1978) 
surveys  data  on  information  input  overload  that  demonstrates  that  the  same  basic  relationship  is 
found  at  all  system  levels.  Specifically,  the  data  demonstrate  that  as  input  to  a  system  increases, 
its  output  initially  increases,  then  decreases.  Many  other  cross-level  generalizations  may  be 
examined  to  assist  our  understanding  of  system  processes.  Table  2  shows  examples  of  these 
subsystem  processes  at  the  level  of  the  organism  and  the  group.  Many  additional  details  may  be 
found  in  Miller  (1978). 

Temporary  Subsystem  Processes 

Subsystems  can  aggregate  temporarily  to  accomplish  a  function,  then  break  up  when  that 
particular  function  has  been  completed.  This  may  occur  numerous  times  during  instructional 
system  development.  In  addition,  individuals  and  groups  may  simultaneously  engage  in  multiple 
subsystem  processes.  It  is  important  that  individuals  engaged  in  ADL  system  development 
understand  who  participates  in  each  subsystem  process  so  information  can  be  efficiently 
transmitted  when  and  where  it  is  needed.  The  next  section  discusses  instructional  systems  and 
how  the  application  of  system  concepts  can  be  applied  to  ADL  systems.  Several  system 
approaches  will  be  then  be  used  to  illustrate  analyses  of  selected  ADL  system  and  subsystem 
functions. 
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Table  2 

Examples  of  the  19  Critical  Subsystems  in  an  Organism  and  a  Group 


[  Subsystems  Which  Process  Both  Matter-Enerj 

yy  and  Information 

Organism 

Group  1 

Reproducer 

Eggs,  sperm,  sex  glands 

Persons  that  produce  implicit  or  explicit  charter 
for  group. 

Boundary 

Epidermis,  fur,  hair,  artifacts  (hat,  coat, 
astronauts  suit) 

Membership  committee,  sergeant-at-arms, 
artifacts  (room,  building,  wall) 

1  Subsystems  Which  Process  Matter-Energy 

|  Subsystems  Which  Process  Information  | 

Group 

Group 

Ingestor 

Mouth,  jaws, 
artifacts  (stomach 
tube,  syringe) 

Refreshment 
committee,  budget 
manager  who 
accepts  fund 
transfers 

Input 

Transducer 

Internal 

Transducer 

Components  of  all 
sensory  modalities 
(e.g.,  eyes,  ears, 
chemoreceptors), 
artifacts  ( radio 
receiver) 
Postsynaptic 
regions  of  neurons 

Lookout,  scout,  artifacts 
(e-mail) 

Subgroup  or  person  who 
receives  information  about 
group  tasks  and  conveys  it 
to  decider 

Distributor 

Blood  &  lymph 
vascular  systems 

Person  who  passes 
out  tools  to  work 
group,  artifacts 
(delivery  truck) 

Channel  and 

Net 

Network  of 
neurons,  hormones 
(conveyed  by 
blood  and  lymph 
systems) 

Each  group  member  who 
communicates  to  other 
members,  artifacts  (written 
messages) 

Converter 

Mouth,  teeth, 
digestive  system 

Chopper  of  wood, 
butcher,  artifacts 
(hand  tools) 

Decoder 

Retinal  bipolar  and 
ganglion  cells, 
cochlear  bipolar 
cells 

Guide,  interpreter,  radar 
man 

Producer 

None  known 

Cook,  tailor, 
maintenance 
technician 

Associator 

Specific 
components  not 
known 

Laterally  dispersed  to 
members  who  associates 
bits  of  information, 
artifacts  (databases) 

Matter- 

Energy 

Storage 

Fatty  tissues,  liver, 
bone  marrow 

Stock  clerk,  spare- 
parts  man 

Memory 

Decider 

Encoder 

Brain  processes 
(specific 
components  not 
known) 

Parts  of  cerebral 
cortex,  artifacts 
(calculator) 
Temporoparietal 
area  of  dominant 
hemisphere  of 
brain 

Secretaries,  treasurers, 
artifacts  (notes,  computer 
files) 

Chairperson,  selected 
specialists 

Persons  composing  letter, 
briefing,  or  statement 
presenting  views  of  group 

Extruder 

Motor 

Supporter 

Kidneys,  rectum, 
breathing 
passages,  sweat 
glands 

Muscles,  bones 
and  joints,  legs, 
artifacts  (cane, 
cart,  automobile) 

Skeleton,  tendons, 
muscles,  artifacts 
(chair,  platform) 

Janitor,  “bouncer" 

Artifacts  (bus, 
truck,  plane),  may 
be  laterally 
dispersed  to 
persons  who  plan 
and  execute  group 
movement 

Person(s) 
supporting  others 
in  group,  artifacts 
(e.g.,  room  car, 
furniture) 

Output 

Transducer 

Exocrine  glands, 
inferior  frontal 
cortex  of  dominant 
hemisphere  of 
human  brain, 
artifacts  (pencil, 
radio  transmitter, 
computer  network) 

Subgroups  or  individuals 
who  deliver  reports  or 
statements  for  the  group 
(spokesman,  publicity 
agent,  chairman) 
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INSTRUCTIONAL  SYSTEMS 
AND  ADL  AS  AN  INSTRUCTIONAL  SYSTEM 

Often  individuals  distinguish  between  training  and  education,  the  former  focusing  on  job 
skills  and  the  latter  on  general  knowledge.  For  the  purposes  of  this  report,  both  of  these 
categories  will  be  included  under  the  term  instructional  systems. 

ISD  AND  SYSTEMS  APPROACHES  TO  TRAINING 

The  objective  of  instructional  systems  is  to  change  behaviors  so  individuals  can  perform  their 
jobs  more  effectively.  Recognizing  the  systemic  nature  of  instruction,  the  military  mandated  use 
of  the  Instructional  Systems  Development  (ISD)  process  (Branson,  Rayner,  Cox,  Furman,  King, 
&  Hannum,  1975).  ISD  is  a  result  of  conceptual  development  that  can  be  traced  to  the 
emergence  of  systems  analysis  during  World  War  II  (Montemerlo  &  Tennyson,  1976). 

Systems  analysis  was  developed  to  solve  problems  associated  with  weapon  systems  that  were 
so  complex  that  their  understanding  "strains  human  comprehension  when  initially  viewed  in  their 
entirety"  (Montemerlo  &  Tennyson,  1976,  p.  9).  Several  efforts  during  the  1950's  and  early 
1960's  attempted  to  apply  systems  analysis  to  training  development  (Miller,  1954;  Kershaw  & 
McKean,  1959;  Hoehn,  1960).  These  and  similar  efforts  were  labeled  systems  approaches  to 
training  (SAT)  and  served  as  the  prototypes  for  the  subsequent  development  of  ISD  (Branson  & 
Grow,  1987).  Unfortunately,  SAT  was  unsuccessful  because  too  many  individuals  assumed  that 
the  labor-intensive  methods  and  models  used  by  training  development  experts  could  be  applied 
by  laymen,  if  they  were  given  simplified  procedural  checklists.  Over  100  proceduralized  SAT 
manuals  were  produced  between  1960  and  1975  (Montemerlo  &  Tennyson,  1976).  These 
manuals  adopted  the  label  SAT,  but  abandoned  the  interdisciplinary  approach  and  methods  of 
SAT.  Furthermore,  many  educational  psychologists  (e.g.,  Campbell,  1971;  Glaser  &  Resnick, 
1972;  McKeachie,  1974)  concluded  that  the  available  theory  and  empirical  evidence  on  learning 
processes  did  not  support  this  over-simplified  approach.  Nevertheless,  many  training  managers 
expected  the  proceduralized  approach  to  work.  When  it  did  not,  they  concluded  that  the  systems 
approach  was  not  an  effective  method  for  the  design  of  instructional  programs  (Montemerlo  & 
Tennyson,  1976).  This  negative  view  of  the  systems  approach  has  made  it  very  difficult  to 
implement  ISD.  A  greater  appreciation  of  the  systemic  nature  of  ISD  will  support  the  use  of 
more  flexible  methodologies  for  ADL  development. 

INSTRUCTIONAL  SYSTEM  PROCESSES 

An  instructional  system  consists  of  the  planned  interaction  of  people,  materials,  and 
techniques,  with  the  goal  of  improved  human  performance  as  measured  by  established  criteria  on 
the  job.  Table  3  lists  some  of  the  types  of  people,  materials,  and  techniques  involved  in 
instructional  system  development.  Each  element  listed  in  Table  3  can  be  viewed  as  part  of  one  or 
more  group  subsystem  processes.  Each  process  is  influenced  by  its  own  perspectives,  interests, 
and  goals,  based  on  its  relationship  to  the  whole  instructional  system.  Furthermore,  like  any 
other  system,  these  subsystems  affect  and  are  effected  by  each  other  to  various  degrees. 
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Table  3 

Some  of  the  Major  Elements  of  an  Instructional  System 


People 

Materials 

Techniques 

•  Students 

•  Training  Aids,  Equipment,  & 

•  Instructional  Requirements 

•  Instructors 

Simulators 

Analysis  Methods 

•  Content  Developers 

•  Instructional  Requirements 

•  Instructional  Design 

•  Instructional  Administrators 

Documents 

Approaches 

•  Subject  Matter  Experts 

•  Instructional  Evaluation 

•  Instructional  Strategies 

•  Training  Aids,  Equipment,  & 

Instruments 

•  Instructional  Development 

Simulator  Developers 

•  Instructional  Development  Tools 

Methods 

•  On-the-job  Supervisors 

•  Instructional  System  Researchers 

(e.g.,  authoring  systems) 

•  Training  Aids,  Equipment,  & 
Simulator  Design  Methods 

•  Training  Effectiveness 

Analysis  Methods 

•  Cost  Effectiveness  Analysis 
Methods 

•  Logistics  Analysis  Methods 

ISD  can  be  depicted  as  a  cyclical  or  iterative  process.  Various  ISD  models  divide  this  process 
into  different  numbers  of  phases  (e.g.,  the  main  DoD  model  includes  five  phases,  Branson,  et  al., 
1975;  a  model  endorsed  by  the  American  Society  for  Training  Development  has  four  phases, 
Rothwell  &  Kazanis,  1998) .  Figure  3  is  a  diagram  of  this  cycle  divided  into  four  phases.  Each 
section  of  the  cycle  influences  every  other  section  either  directly  or  indirectly,  through  the 
interactions  of  people,  materials,  and  techniques  (as  listed  in  Table  3).  It  is  also  important  to 
note  that  external  factors  (e.g.,  technological  developments,  policy  constraints,  and  changes  in 
job  requirements)  also  are  powerful  influences  in  this  cycle. 

The  instructional  development  cycle  may  be  further  divided  into  subsystem  activities 
supporting  each  ISD  phase  (see  Figure  4).  As  mentioned  above,  many  of  these  subsystem 
activities  are  time-dependant  processes,  which  end  when  their  immediate  goal  is  achieved  and 
their  product  is  passed  on  to  another  subsystem  activity.  For  example,  the  selection  of  an 
instructional  strategy  is  usually  completed  early  in  the  developmental  phase  based  on  inputs  from 
other  activities,  such  as  the  development  of  instruction  by  objectives.  If  the  instructional 
objective  is  mastery  of  a  motor  skill,  the  instructional  strategy  will  probably  include  guided 
practice  and  feedback.  Throughout  the  ISD  process,  these  subsystem  activities  are  supported  and 
influenced  by  previous  activities  and  also  support  and  influence  subsequent  activities.  Figure  4 
is  oversimplified;  it  does  not  depict  multiple  instances  of  activities  in  different  or  multiple 
developmental  phases.  For  example,  formative  evaluations  may  be  conducted  during  the 
program  development  phase  as  well  as  during  program  implementation. 
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Figure  3.  Phases  of  the  Instructional  Development  Cycle 


ADVANCED  DISTRIBUTED  LEARNING  (ADL) 

Many  educators,  program  managers,  and  instructional  developers  believe  that  ADL  will  be  a 
major  training  tool  in  the  21st  century.  The  Department  of  Defense  (DoD)  provides  the 
following  definitions  in  its  ADL  Implementation  Plan  (DoD,  2000): 

Learning  is  defined  as  the  acquisition  of  knowledge,  skills,  behaviors,  and 
attitudes  (through  the  integration  of  education,  training,  and  performance  support 
in  a  comprehensive,  mutually  supportive  system). 

Distributed  Learning  ( encompassing  programs  also  referred  to  as  distance 
learning)  is  defined  as  structured  learning  that  takes  place  without  requiring  the 
physical  presence  of  an  instructor.  Distributed  learning  is  synchronous  and/or 
asynchronous  learning  mediated  with  technology  and  may  use  one  or  more  of  the 
following  media:  audio/videotapes,  CD-ROMs,  audio/videoteletraining, 
correspondence  courses,  interactive  television,  and  video  conferencing. 

Advanced  Distributed  Learning  is  an  evolution  of  distributed  learning  (distance 
learning)  that  emphasizes  collaboration  on  standards-based  versions  of  reusable 
objects,  networks,  and  learning  management  systems,  yet  may  include  some 
legacy  methods  and  media,  (p.  ES-2;  emphasis  in  original) 
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Figure  4.  Selected  Instructional  Development  Subsystem  Processes 
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Although  a  different  medium  of  instruction,  ADL  systems  operate  under  the  same  system 
constraints  as  any  other  type  of  instructional  system.  Therefore  the  ISD  and  system  concepts 
discussed  above  can  be  beneficially  applied  to  ADL  systems. 
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SYSTEM  APPROACHES  APPLIED  TO  ADL 

System  theories  provide  a  variety  of  tools  to  analyze  and  improve  ADL  processes.  Equally 
important  as  these  tools,  a  system  perspective  changes  the  way  one  views  the  world.  "A  large 
part  of  the  battle  is  getting  the  concept  accepted. .  .so  that  they  appreciate  the  interactions  of 
members  of  a  system  and  the  environment,  and  to  be  on  the  watch  for  such  interactions"  (Rubin, 
1971,  p.  11). 

* 

CONSIDERATIONS  TO  HELP  UNDERSTAND  ADL  SYSTEMS 

Churchman  (1968)  identified  five  basic  considerations,  which  help  individuals  understand 
important  aspects  of  systems.  These  considerations  will  serve  as  the  orientation  for  the  following 
discussion  of  ADL  systems.  Selected  system  approaches  will  be  applied  to  illustrate  how  they 
can  help  reorganize  complex  ADL  processes  into  forms  that  are  easier  to  understand  (Weinberg, 
1975). 

Total  ADL  System  Objectives 

Churchman's  (1968)  first  consideration  is  to  be  aware  of  the  total  system  objectives  and  more 
specifically,  the  performance  measures  of  the  whole  system.  It  is  often  difficult  to  determine  the 
objectives  of  the  total  system  because  most  individuals  do  not  realize  that  their  subsystem  is  not 
'  autonomous.  Once  this  realization  occurs,  the  focus  of  analysis  can  be  expanded  to  the  network 
of  sub-  and  suprasystems  that  affect  the  local  subsystem. 

To  analyze  the  total  ADL  system  goals,  one  must  first  examine  the  sub-  and  suprasystem 
relationships.  Figure  5  is  a  very  simplified  diagram  showing  some  of  the  relationships  between 
several  DoD  organizations  involved  in  ADL.  The  Office  of  the  Secretary  of  Defense  (OSD)  is 
shown  on  the  top  left  of  Figure  5.  OSD  has  the  responsibility  for  executing  the  ADL  Initiative  to 
help  actualize  the  vision  summarized  by  the  Secretary  of  Defense.  "The  Department  of  Defense's 
vision  is  to  ensure  that  Department  of  Defense  personnel  have  access  to  the  highest  quality 
education  and  training  that  can  be  tailored  to  their  needs  and  delivered  cost  effectively,  anytime 
and  anywhere”  (Cohen,  1999,  quoted  in  DoD,  2000,  cover  materials).  This  primary  objective 
governs  the  efforts  of  the  organizations  shown  in  Figure  5  to  varying  degrees.  As  the  following 
discussion  demonstrates,  each  of  these  organizations  has  its  own  responsibilities,  which  may  lead 
to  conflicts  in  realizing  the  primary  objective. 

OSD  has  established  the  ADL  Co-Laboratory  network  (shown  on  the  left  side  of  Figure  5). 
This  network,  as  reflected  by  its  title,  fosters  collaborative  efforts  to  advance  the  ADL  Initiative. 

*  It  consists  of  the  ADL  Co-Laboratory  whose  "most  critical  function  is  to  develop,  evaluate,  and 

promote  ADL  standards"  (DoD,  2000,  p.  ES-3).  Two  Co-Lab  nodes,  the  Academic  Co-Lab  and 
the  Joint  Co-Lab,  have  additional  objectives.  The  Academic  ADL  Co-Lab  promotes  the 
development  of  ADL  technologies,  principally  among  academic  institutions.  The  Joint  ADL  Co- 
Lab  "was  established  to  promote  collaborative  and  rapid  development  of  ADL  prototypes  and 
ADL  system  acquisitions"  (DoD,  2000,  p.  ES-3).  This  goal  brings  the  Joint  ADL  Co-Laboratory 
into  a  direct  relationship  with  some  of  the  organizations  charged  with  implementation  of  ADL. 
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^Academic  Co-LabN 
(academic  outreach), 


'  Joint  Co-Lab 
(Implementation^ 


Implementation 

Interactions 


Note.  OSD=Office  of  the  Secretary  of  Defense; 
NAVAIR=Naval  Air  Systems  Command; 
AFIADL=Air  Force  Institute  for  ADL; 
ATSC=  Army  Training  Support  Center; 
PMA=Program  Manager  Aviation 


Figure  5.  Example  Relationships  Among  Some  ADL  Organizations 
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A  small  example  of  the  many  organizations,  which  implement  ADL  is  shown  on  the  right  side 
of  Figure  5.  This  simplified  view  depicts  three  service  specific  organizations:  the  Naval  Air 
Systems  Command  (NAVAIR),  the  Air  Force  Institute  for  ADL,  and  the  Army  Training  Support 
Center  (ATSC).  Each  of  these  organizations  has  different  degrees  of  responsibility  for  the  ADL 
training  being  implemented  in  the  Navy,  Air  Force,  and  Army.  Due  to  space  limitations,  this 
discussion  will  focus  on  NAVAIR,  but  it  should  be  noted  that  similar  activities  and  organizations 
are  found  in  the  other  services. 

NAVAIR  is  the  Navy's  principal  command  for  the  development,  acquisition,  and  support  of 
naval  aeronautical  and  related  technology  systems.  The  acquisition  of  aeronautical  training 
systems  is  accomplished  by  PMA-205.  ADL  systems  are  part  of  this  responsibility.  As  such, 
PMA-205  is  in  the  process  of  converting  thousands  of  legacy  courses  for  delivery  on  the  internet, 
as  well  as  developing  new  ADL  courses.  It  must  accomplish  this  on  a  tight  schedule  and  may 
seek  implementation  advice  from  the  Joint  ADL  Co-Lab. 

Here  is  an  example  of  where  a  goal  conflict  may  arise.  The  Joint  ADL  Co-Lab,  as  part  of  the 
ADL  Co-Lab  network,  fully  supports  the  development  of  ADL  standards,  embodied  in  the 
Sharable  Content  Object  Reference  Model  (SCORM).  SCORM  is  being  developed  as  a 
multiyear  effort,  with  the  participation  of  various  industry  associations  and  professional 
organizations.  SCORM  development  is  proceeding  through  various  versions,  each  more 
comprehensive  than  the  last.  Version  1.0  was  released  in  January,  2000  and  version  1.1  was 
released  in  January,  2001.  Version  2.0  is  scheduled  to  be  released  during  2001  and  subsequent 
versions  may  follow. 

The  Joint  ADL  Co-Lab  is  also  tasked  with  assisting  in  the  implementation  of  ADL  in  the 
military  training  acquisition  community.  PMA-205  seeks  immediate  assistance  to  help  them 
conform  to  SCORM  while  meeting  their  course  conversion  and  development  schedule. 
Unfortunately,  SCORM  is  an  evolving  standard  and  does  not  currently  support  important 
instructional  requirements.  For  example,  SCORM  does  not  include  a  mechanism  for  passing  the 
results  of  performance  testing  from  one,  stand-alone  learning  object  to  another.  To  be 
conformant  with  the  current  SCORM,  instructional  developers  either  need  to  wait  for  more 
comprehensive  versions  or  recommend  suboptimal  instructional  strategies.  The  latter  is  not  an 
acceptable  solution,  so  the  Joint  Co-Lab  is  forced  to  recommend  compromises  on  SCORM  to 
meet  PMA-205's  short-term  schedule.  A  plan  for  content  development  in  the  "spirit  of  SCORM" 
(e.g.,  organizing  content  into  small  stand-alone  learning  objects  and  later  tagging  these  content 
objects  with  conformant  SCORM  meta-data)  may  enable  DoD  organizations  to  meet  both  sets  of 
goals.  Many  other  examples  of  goal  conflicts  can  be  identified  and  resolved  through  a  total- 
system  goal  analysis. 

Understanding  the  ADL  System's  Environment 

The  second  of  Churchman's  considerations  is  to  understand  the  system's  environment — the 
fixed  constraints  that  affect  the  system.  One  of  the  fixed  constraints  on  an  ADL  system  is  the 
channels  that  bring  information  into  and  out  of  the  system.  In  LST  terminology,  the  subsystem 
that  brings  information  into  a  system  is  called  the  input  transducer  (Miller,  1978).  This  function 
may  be  upwardly  dispersed  to  the  suprasystem  (e.g.,  tasking  memoranda  or  press  releases), 
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laterally  dispersed  to  multiple  individuals  in  the  system  (e.g.,  attendees  at  policy  meetings), 
downwardly  dispersed  to  one  or  more  subsystems  (e.g.,  individuals  obtaining  data  to  support  a 
specific  subprocess),  or  various  combinations.  This  dispersion  of  the  input  transducer  function 
may  result  in  conflicting  or  delayed  information  required  by  other  subsystem  processes. 

Another  important  aspect  of  the  analysis  of  an  ADL  system's  informational  constraints,  is  to 
search  for  sources  of  information  distortion.  Such  distortion  can  be  due  to  a  variety  of  conditions 
including  information  overload.  However,  in  organizations,  the  distortion  of  information  is  often 
due  to  biases  introduced  by  components  evaluating  information  from  a  local  rather  than  a 
system-wide  perspective.  Individuals  may  report  what  they  believe  their  superiors  want  to  hear 
or  what  they  want  them  to  hear  (Guetzkow,  1965;  Katz  &  Kahn,  1966).  Identification  and 
removal  of  sources  of  information  distortion  will  enhance  organizations'  abilities  to  advance  the 
ADL  agenda. 

Environmental  constraints  also  include  the  external  forces  that  influence  ADL  system 
development.  Some  of  these  forces  are  shown  in  Figure  3;  they  include: 

•  Technological  developments,  which  influence  how  instructional  content  may  be 
delivered  to  students.  For  example,  delivering  web-based  instruction  to  students  aboard  ship  is 
constrained  by  available  bandwidth.  As  new  technologies  expand  bandwidth,  additional 
instructional  options  (e.g.,  streaming  video)  become  available. 

•  Administrative  and  policy  constraints  influence  how  ADL  content  will  be  developed 
and  delivered.  For  example,  delivery  of  instruction  over  the  web  has  security  implications  that 
must  be  understood  by  instructional  developers  and  may  influence  the  instructional  strategy 
chosen. 


•  Job  changes  must  be  constantly  monitored  to  ensure  that  ADL  instruction  is  current, 
providing  the  latest,  most  accurate  information. 

Understanding  the  Resources  Available 

Churchman’s  (1968)  third  consideration  is  to  understand  the  resources  available  to  the  ADL 
system.  Instructional  developers  are  often  asked  to  do  more  with  less.  However,  it  is  important 
that  a  comprehensive  analysis  of  the  resource  requirements  be  conducted  to  determine  whether 
the  goals  of  an  ADL  program  can  be  achieved.  If  sufficient  resources  are  not  available,  or  if 
resources  are  spread  across  too  many  efforts,  alternate  plans  should  be  formulated. 

Understanding  ADL  Subsystem  Components 

One  should  be  aware  of  the  components  (subsystems)  of  the  ADL  system  in  terms  of  their 
activities,  goals,  and  measures  of  performance.  Organizing  the  analysis  using  Miller's  (1978) 
critical  subsystem  processes  can  provide  a  useful  starting  point.  Each  subsystem  process  (see 
Tables  1  and  2)  should  be  examined  as  it  relates  to  all  other  processes.  For  example,  the 
associator  is  the  subsystem  that  carries  out  the  first  stage  in  the  learning  process  of  the  system. 

To  form  enduring  associations,  the  associator  must  receive  information  from  the  input  transducer 


26 


Technical  Report  2001-006 


and  the  internal  transducer  via  the  channel  and  net.  It  must  then  transfer  this  information  into 
memory  for  longer-term  storage.  An  ADL  subsystem  analysis  needs  to  identity  the  individuals, 
groups,  and  organizations  that  carry  out  these  and  other  subsystem  processes.  Next,  the  specific 
information  channels  and  decision  points  can  be  identified  and  evaluated.  Finally,  the 
impediments  to  successfully  reaching  subsystem  goals  in  each  information  channel  can  be 
removed  or  reduced. 

*  Understanding  the  Management  of  the  ADL  System 

.  The  fifth  consideration  for  understanding  ADL  systems  is  to  identify  and  understand  the 

management  of  the  system.  This  includes  the  constraints  imposed  upon  it  by  its  suprasystem  or 
multiple  suprasystems  (see  Figures  1  and  2).  In  ADL  systems,  these  constraints  include  some  of 
those  discussed  above  (e.g.,  goal  conflicts,  lack  of  resources,  and  information  distortion).  A 
more  subtle  constraint,  related  to  information  distortion,  is  the  suprasystem  perception  that  goals 
have  been  achieved.  Because  the  suprasystem's  information  is  more  general  and  may  be 
distorted  by  subsystem  inputs,  it  is  easy  to  overestimate  the  success  or  degree  of  completion  of 
subsystem  efforts.  This  could  be  the  result  of  imprecise  or  improper  measures  of  performance. 

It  could  also  be  the  result  of  the  suprasystem  overselling  results  to  its  controlling  suprasystem.  It 
is  equally  important  for  the  suprasystem  components  to  understand  system  relationships  and  the 
constraints  it  imposes  upon  its  subsystems  as  it  is  for  the  subsystem  components  to  recognize  the 
constraints  imposed  by  their  suprasystems. 


f 
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ADL  SYSTEM  RESEARCH  AGENDA 

A  system  perspective  can  help  ADL  researchers  identify  areas,  which  will  generate  benefits 
for  the  total  ADL  system.  At  a  minimum,  the  following  topics  require  additional  research. 

APPLY  SYSTEM  CONCEPTS  AND  APPROACHES  TO  ADL 

The  above  discussions  have  provided  simple  examples  of  some  system  concepts  (e.g.,  the  19 
critical  subsystems)  and  approaches  (e.g.,  information  flow  analysis).  Researchers  need  to 
conduct  more  in-depth  application  of  these  concepts  and  tools  in  the  specific  ADL  system 
context.  Each  ADL  effort  should  be  analyzed  in  terms  of  the  critical  subsystem  processes. 

Which  individuals  or  groups  are  involved  in  each  process  or  multiple  processes?  Are  they  aware 
of  their  involvement  in  a  given  process?  What  are  their  resources  and  constraints?  How  do  they 
communicate  to  other  processes?  The  answers  to  these  and  many  other  questions  are  required 
before  meaningful  actions  may  be  taken  to  improve  the  quality  of  the  ADL  efforts. 

In  conducting  a  system  analysis  of  ADL  efforts,  special  emphasis  should  be  placed  on  the 
interfaces  between  various  subsystems  and  subsystem  processes.  Information  and  other  products 
from  each  subsystem  process  pass  to  other  systems  or  subsystem  processes  across  these 
interfaces.  It  is  at  the  interfaces  that  problems  often  occur  (e.g.,  information  overload  or 
information  distortion).  The  structures  and  functions  that  help  or  hinder  transfer  across 
subsystem  interfaces  can  be  analyzed  and  improved. 

CROSS-LEVEL  GENERALIZATIONS 

A  large  body  of  empirical  data  demonstrates  that  many  cross-level  generalizations  can  guide 
our  analysis  of  systems  (Miller,  1978).  Researchers  can  identify  and  apply  these  to  support  ADL 
analyses.  For  example,  at  the  cell  level,  the  boundary  function  includes  the  semi-permeable  cell 
membrane.  This  semi-permeability  allows  certain  substances  to  pass  into  the  cell  while  blocking 
others.  It  is  likely  that  an  analogous  process  occurs  in  ADL  systems.  Certain  types  of 
information  are  recognized  and  acted  on  by  the  ADL  system  and  other  types  are  blocked  or 
ignored.  A  detailed  analysis  of  the  ADL  system  boundary  will  help  identify  problems  and 
improve  information  flow.  Many  other  cross-level  generalizations  can  be  identified  and  used  to 
focus  ADL  system  analysis  efforts. 

IDENTIFICATION  OF  ADL  SYSTEM  RELATIONSHIPS 

It  is  vital  to  understand  the  scope  of  ADL  efforts.  Many  organizations  are  conducting 
collaborative  ADL  efforts  (e.g.,  the  ADL  Co-Lab  Network).  However,  many  other  organizations 
have  their  own  ADL  agendas,  sponsors,  and  constraints.  A  comprehensive  survey  of  ADL 
activities  will  help  identify  projects,  which  duplicate  or  compete  with  one  another.  It  can  also 
identify  where  areas  of  collaboration  exist  and  can  be  expanded. 
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ADL  GOALS  AND  NEEDS  ANALYSIS 

Once  ADL  organizations  and  their  relationships  have  been  identified,  the  focus  can  be  shifted 
to  an  analysis  of  each  organization's  goals  and  needs.  A  systems  perspective  helps  to  illuminate 
possible  goal  conflicts  between  various  subsystem  activities.  Identification  and  reduction  of 
these  conflicts  will  enhance  collaboration  in  the  total  ADL  development  process. 

Comprehensive  ADL  needs  analysis  requires  tools  and  methods  to  identify  the  goals  and 
communication  mechanisms  of  each  ADL  sub-  and  suprasystem.  Special  emphasis  should  be 
given  to  how  the  suprasystem  communicates  its  goals  to  its  subsystems  and  how  these  goals  are 
interpreted  at  the  subsystem  level.  It  is  also  important  to  identify  how  the  goals  and  needs  of 
individuals  working  on  one  subsystem  process  are  communicated  to  individuals  engaged  in  other 
subefforts.  The  identification  and  communication  of  goals  and  needs  across  subsystems  will 
help  avoid  duplication  of  effort  and  enhance  the  use  of  one  subsystem's  products  by  individuals 
working  in  other  areas. 

ADL  DEVELOPMENT  GUIDELINES 

Since  ADL  is  a  new  instructional  approach,  ADL  developers  need  the  best  possible  guidance 
to  help  them  achieve  their  goals.  Some  guidance  can  be  found  in  the  Human-Computer  Interface 
(HCI)  literature  or  the  literature  on  computer-based  instruction  (e.g.,  choice  of  font  size  or  use  of 
colors  on  computer  screens).  Other  guidance  is  specific  to  ADL  instruction  (e.g.,  when  to  use  e- 
mail  or  chat  rooms  as  part  of  an  ADL  course).  The  results  of  an  ongoing  effort  to  accumulate 
ADL  design  and  evaluation  guidelines  (Hamel,  Ryan-Jones,  &  Hays,  2000)  will  be  of  greater 
utility  if  they  are  organized  from  a  total  ADL  system  perspective.  Guidelines  may  become  too 
narrowly  focused  on  a  limited  number  of  subprocesses  at  the  expense  of  others.  Researchers 
who  are  developing  or  accumulating  and  validating  ADL  guidelines  must  be  aware  that 
providing  guidance  to  individuals  working  on  one  subprocess  will  not  guarantee  effective  ADL 
systems  if  individuals  engaged  in  other  subprocesses  lack  guidance. 

The  ISD  process  (e.g..  Figure  3)  can  provide  the  initial  organization  for  the  guidelines.  Each 
guideline  should  address  HCI  (appearance  and  usability)  and  informational  (instructional 
content)  issues.  This  is  similar  to  the  analysis  of  physical  and  functional  fidelity  requirements 
used  by  many  simulator  developers  to  guide  their  efforts  (Hays  &  Singer,  1989).  Like  physical 
and  functional  fidelity,  the  HCI  and  informational  components  are  analogous  to  the  visual 
components  of  brightness  and  hue.  Neither  can  be  changed  without  some  degree  of  change  in  the 
other.  Both  aspects  must  be  addressed  to  ensure  that  ADL  decisions  do  not  result  in 
counterproductive  effects. 

It  is  also  necessary  to  target  ADL  guidelines  toward  the  various  disciplines  that  undertake 
various  ISD  subprocesses.  For  example,  some  guidelines  should  be  targeted  toward  the 
instructional  developers  and  others  toward  the  web  programmers.  Without  focusing  the  ADL 
guidelines  on  the  requirements  of  individuals  from  various  disciplines,  they  may  not  be  useful  to 
the  people  who  need  them  during  each  ISD  phase. 

ADL  guidelines  should  be  based  on  empirical  research,  especially  those  that  address 
instructional  strategies.  Although  data  on  the  effectiveness  of  instruction  has  been  collected  in 
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classrooms  (e.g.,  Bloom,  1984)  and  in  the  context  of  training  simulation  (e.g.,  Hays  &  Vincenzi, 
2000),  there  is  a  dearth  of  data  on  the  effectiveness  of  ADL  applications.  As  ADL  guidelines  are 
accumulated  and  applied,  gaps  in  the  empirical  foundation  for  the  guidelines  can  be  identified 
and  used  to  identify  and  justify  ADL  research  agendas.  Data  from  this  research  may  then  be 
used  to  make  informed  decisions  on  the  selection  of  ADL  instructional  strategies. 

EVALUATION  OF  ADL  SYSTEMS 

• 

Instructional  system  evaluation  is  always  difficult.  Transfer  of  Training  (TOT)  has  been 
advocated  as  the  most  valid  method  for  measuring  instructional  effectiveness  (Orlansky,  1981) 
and  several  TOT  models  and  evaluation  designs  have  been  developed  (Caro,  1977;  Pfeiffer  & 
Browning,  1984;  Hays  &  Singer,  1989).  However,  TOT  is  seldom  used  because  of  numerous 
factors  (e.g.,  lack  of  resources  to  track  students  after  training  and  measure  their  performance  at 
the  job  site).  Cost  and  scheduling  factors  may  also  be  used  to  evaluate  instructional  systems 
(Orlansky  &  String,  1981).  Often  the  type  of  evaluation  is  mandated  by  whichever  sub-  or 
suprasystem  generates  the  requirements  or  provides  funds  for  the  evaluation. 

Determining  the  effectiveness  of  ADL  systems  is  more  difficult  than  some  other  instructional 
systems  because  students  can  access  different  types  of  instruction  or  job  aiding  information  from 
diverse  locations  and  it  would  be  prohibitively  costly  to  collect  performance  data  at  each 
location.  Tools  and  methods  are  needed  to  ensure  that  evaluation  techniques  address  the 
divergent  requirements  of  various  ADL  organizations  and  activities.  Some  evaluations  may  need 
to  focus  on  the  cost  of  ADL  development  and  implementation;  others  on  the  number  of  students 
that  have  access  to  instruction  or  the  number  of  instructors  or  facilitators  necessary  to  support 
ADL  systems.  In  most  cases,  improvements  in  job  performance  should  be  a  primary  goal  of 
ADL  evaluations.  Since  no  evaluation  can  meet  all  goals,  identification  and  prioritization  of  sub- 
and  suprasystem  goals  can  help  allocate  evaluation  resources  for  the  greatest  total  ADL  system 
benefit. 

CONFIGURATION  MANAGEMENT  OF  ADL  SYSTEMS 

Managers  of  instructional  systems  that  use  simulators  have  long  been  aware  of  the  critical 
need  for  configuration  management.  This  includes  the  need  to  upgrade  simulator  software  as 
new  versions  are  released.  ADL  systems  present  a  new  set  of  configuration  management  issues. 
For  example,  if  web-based  instruction  uses  links  to  other  web  sites,  someone  needs  to  monitor 
each  link  to  ensure  that  it  is  operational.  Furthermore,  it  is  necessary  to  ensure  that  the 
f  information  on  the  web  sites  that  are  linked  to  an  ADL  application  is  current  and  continues  to 

support  its  instructional  needs.  System  techniques  and  methods  can  help  identify  the  critical 
/  ADL  areas  requiring  configuration  management. 
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SUMMARY  AND  RECOMMENDATIONS 

Effective  ADL  systems  can  become  a  potent  tool  to  provide  the  necessary  knowledge  and 
skills  for  individuals  to  compete  in  an  increasingly  complex  world.  This  report  has  presented  a 
theoretical  foundation  using  a  system  perspective  to  aid  in  the  development  of  a  research  agenda 
for  ADL  systems.  System  theory  focuses  on  the  complex  interactions  within  systems  and  instills 
a  paradigm  shift  in  the  way  individuals  engaged  in  the  development  of  ADL  systems  approach 
*  their  task.  After  a  brief  review  of  system  theories,  concepts  and  approaches,  selected  system 

concepts  were  discussed  in  the  context  of  the  development  and  evaluation  of  ADL  systems. 

,  Guided  by  the  system  perspective,  the  application  of  system  tools  and  concepts  can  improve  our 

ability  to  develop  effective  and  economical  ADL  systems.  Several  ADL  research  areas  and 
issues  were  discussed  and  recommendations  for  system-oriented  research  were  provided. 

It  is  recommended  that  all  individuals  engaged  in  the  development  of  ADL  policy  and 
products  adopt  a  total  system  perspective.  This  perspective  will  ensure  that  individuals  focus 
their  energy  and  attention  on  several  important  areas: 

•  Total  system  and  subsystem  relationships,  to  help  develop  an  understanding  of  all  of  the 
various  systems  that  engage  in  ADL  efforts  and  how  they  can  support  one  another. 

•  System  and  subsystem  goals  and  needs,  to  facilitate  the  development  of  shared  goals, 
rather  than  wasting  efforts  on  conflicting  goals. 

•  Communication  processes  between  systems  and  subsystems,  to  help  establish  clear  lines 
of  communication,  reduce  information  distortion,  and  enable  individuals  to  engage  in 
collaborative  efforts. 

•  Apply  system-oriented  tools  and  methods,  to  generate  data  that  will  support  the  total 
ADL  system.  These  data,  applied  across  the  entire  ADL  system,  will  result  in  synergy  and  the 
rapid  advancement  of  the  instructional  capabilities  that  are  potentially  available  from  ADL. 
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